Quantifying motion in 3D is important for studying the behavior of humans and other animals, but manual pose annotations are expensive and time-consuming to obtain. Self-supervised keypoint discovery is a promising strategy for estimating 3D poses without annotations. However, current keypoint discovery approaches commonly process single 2D views and do not operate in the 3D space. We propose a new method to perform self-supervised keypoint discovery in 3D from multi-view videos of behaving agents, without any keypoint or bounding box supervision in 2D or 3D. Our method uses an encoder-decoder architecture with a 3D volumetric heatmap, trained to reconstruct spatiotemporal differences across multiple views, in addition to joint length constraints on a learned 3D skeleton of the subject. In this way, we discover keypoints without requiring manual supervision in videos of humans and rats, demonstrating the potential of 3D keypoint discovery for studying behavior.

现实世界的行为通常是由多种代理之间复杂的相互作用来塑造的。为了可靠地研究多代理行为，无监督和自我监督的学习的进步使从轨迹数据中学到了各种不同的行为表示。迄今为止，还没有一组统一的基准测试，可以在广泛的行为分析设置中进行定量和系统地比较方法。我们的目的是通过引入来自现实世界行为神经科学实验的大规模，多代理轨迹数据集来解决这一问题，该数据集涵盖了一系列行为分析任务。我们的数据集由来自通用模型生物的轨迹数据组成，其中有960万帧的小鼠数据和440万帧的飞行数据，在各种实验环境中，例如不同的菌株，相互作用的长度和光遗传学刺激。框架的子集还包括专家注销的行为标签。我们数据集的改进对应于跨多种生物的行为表示，并能够捕获常见行为分析任务的差异。

我们提出了一种学习来自未标识的行为视频的代理的姿势和结构的方法。从观察开始，表现代理通常是行为视频中的主要运动来源，我们的方法使用具有几何瓶颈的编码器 - 解码器架构来重建视频帧之间的差异。只要仅关注运动区域，我们的方法直接在输入视频上工作，而无需手动注释，例如关键点或边界框。关于各种代理类型（鼠标，飞，人，水母和树木）的实验展示了我们的方法的一般性，并揭示了我们发现的关键点代表着语义有意义的身体部位，这在关键点回归上实现了最先进的性能在自我监督的方法中。此外，我们发现的关键点可实现可比的性能，以对下游任务的监督关键点，例如行为分类，表明我们的方法可以大大降低模型培训VIS-VIS监督方法的成本。

We present a framework for the unsupervised learning of neurosymbolic encoders, which are encoders obtained by composing neural networks with symbolic programs from a domain-specific language. Our framework naturally incorporates symbolic expert knowledge into the learning process, which leads to more interpretable and factorized latent representations compared to fully neural encoders. We integrate modern program synthesis techniques with the variational autoencoding (VAE) framework, in order to learn a neurosymbolic encoder in conjunction with a standard decoder. The programmatic descriptions from our encoders can benefit many analysis workflows, such as in behavior modeling where interpreting agent actions and movements is important. We evaluate our method on learning latent representations for real-world trajectory data from animal biology and sports analytics. We show that our approach offers significantly better separation of meaningful categories than standard VAEs and leads to practical gains on downstream analysis tasks, such as for behavior classification.

多代理行为建模旨在了解代理之间发生的交互。我们从行为神经科学，Caltech鼠标社交交互（CALMS21）数据集中提供了一个多代理数据集。我们的数据集由社交交互的轨迹数据组成，从标准居民入侵者测定中自由行为小鼠的视频记录。为了帮助加速行为研究，CALMS21数据集提供基准，以评估三种设置中自动行为分类方法的性能：（1）用于培训由单个注释器的所有注释，（2）用于风格转移以进行学习互动在特定有限培训数据的新行为学习的行为定义和（3）的注释差异。 DataSet由600万个未标记的追踪姿势的交互小鼠组成，以及超过100万帧，具有跟踪的姿势和相应的帧级行为注释。我们的数据集的挑战是能够使用标记和未标记的跟踪数据准确地对行为进行分类，以及能够概括新设置。

Statistical risk assessments inform consequential decisions such as pretrial release in criminal justice, and loan approvals in consumer finance. Such risk assessments make counterfactual predictions, predicting the likelihood of an outcome under a proposed decision (e.g., what would happen if we approved this loan?). A central challenge, however, is that there may have been unmeasured confounders that jointly affected past decisions and outcomes in the historical data. This paper proposes a tractable mean outcome sensitivity model that bounds the extent to which unmeasured confounders could affect outcomes on average. The mean outcome sensitivity model partially identifies the conditional likelihood of the outcome under the proposed decision, popular predictive performance metrics (e.g., accuracy, calibration, TPR, FPR), and commonly-used predictive disparities. We derive their sharp identified sets, and we then solve three tasks that are essential to deploying statistical risk assessments in high-stakes settings. First, we propose a doubly-robust learning procedure for the bounds on the conditional likelihood of the outcome under the proposed decision. Second, we translate our estimated bounds on the conditional likelihood of the outcome under the proposed decision into a robust, plug-in decision-making policy. Third, we develop doubly-robust estimators of the bounds on the predictive performance of an existing risk assessment.

Realistic synthetic image data rendered from 3D models can be used to augment image sets and train image classification semantic segmentation models. In this work, we explore how high quality physically-based rendering and domain randomization can efficiently create a large synthetic dataset based on production 3D CAD models of a real vehicle. We use this dataset to quantify the effectiveness of synthetic augmentation using U-net and Double-U-net models. We found that, for this domain, synthetic images were an effective technique for augmenting limited sets of real training data. We observed that models trained on purely synthetic images had a very low mean prediction IoU on real validation images. We also observed that adding even very small amounts of real images to a synthetic dataset greatly improved accuracy, and that models trained on datasets augmented with synthetic images were more accurate than those trained on real images alone. Finally, we found that in use cases that benefit from incremental training or model specialization, pretraining a base model on synthetic images provided a sizeable reduction in the training cost of transfer learning, allowing up to 90\% of the model training to be front-loaded.

由于行动和状态空间的连续性，策略的多模式，环境中的障碍的存在以及对其他代理的瞬时适应需要，因此协作式携带是一项复杂的任务。在这项工作中，我们提出了一种预测合作人类手机团队的现实运动计划的方法。使用变性复发性神经网络VRNN来对人类机器人团队的轨迹进行建模，随着时间的流逝，我们能够捕获团队未来状态的分布，同时利用交互历史的信息。我们方法的关键是我们模型利用人类示范数据并产生在测试期间与人协同良好的轨迹的能力。我们表明，与基线，基于集中抽样的计划者快速探索的随机树（RRT）相比，该模型会产生更多类似人类的运动。此外，我们通过人类合作伙伴评估了VRNN规划师，并显示出比RRT在与人类计划时能够产生更类似人类的路径并获得更高的任务成功率的能力。最后，我们证明了使用VRNN规划师使用的Lotobot可以通过控制另一个Locot的人来成功完成任务。

跌倒是致命和非致命伤害的主要原因，尤其是对于老年人。身体内部原因（例如疾病）或外部原因（例如主动或被动扰动）可能导致不平衡。主动扰动是将外力施加到人的结果，而被动扰动是由于人类运动与静态障碍相互作用而导致的。这项工作提出了一个指标，该指标允许监视躯干及其与主动和被动扰动的相关性。我们表明，躯干摇摆的巨大变化可以与主动扰动密切相关。我们还表明，通过调节过去的轨迹，躯干运动和周围场景的预期路径和躯干摇摆，我们可以合理地预测躯干摇摆的未来路径和预期变化。这将有直接的预防应用程序。结果表明，躯干摇摆与扰动密切相关。而且我们的模型能够利用全景图中介绍的视觉提示并相应地调节预测。

协作机器人将对家庭服务应用中的人类福利产生巨大影响，而高级制造业中的工业优势需要灵巧的组装。出色的挑战是为机器人指尖提供一种物理设计，使他们擅长执行需要高分辨率，校准形状重建和力传感的灵活任务。在这项工作中，我们提出了Densetact 2.0，这是一种能够可视化柔软指尖的变形表面并在神经网络中使用该图像来执行校准形状重建和6轴扳手估计的光学传感器。我们证明了用于形状重建的每个像素0.3633mm的传感器精度，0.410N的力量，扭矩为0.387mmnm，以及通过转移学习来校准新手指的能力，实现了可比性的性能，训练了四倍以上，只有12％以上数据集大小。